HPC in Biomedical Research

Overview

Manuel Holtgrewe

Berlin Institute of Health at Charité

Course Overview

  • Welcome to the course! 👋
  • Introduction to High-Performance Computing (HPC)
  • Focus on biomedical and medical research applications

🔬 ⌨️ 🧬

  • Duration: 5 days
  • Instructor: Manuel Holtgrewe
  • Contact Information: manuel.holtgrewe@bih-charite.de

Course Structure

  • Monday to Thursday
    • 8:30-10:00 – Seminar
    • 10:30-15:15 – “Open lab” (incl. 1h lunch break)
  • Friday:
    • 14:00-15:30 – Presentations

Participant Presentations

  • short (5min) presentation with slides on how you plan what you have learned in your PhD project.
  • short (5min) discussion

Course Objectives

  • Fundamentals of HPC …
  • … for biomedical research
  • Practical skills in
    • Linux command line
    • HPC job submission
    • Scientific programming
  • Parallel computing techniques and their applications

Participant Background 🤸

  • Briefly introduce yourself:
    • Name
    • Background (biomedical/computational, programming experience, etc.)
    • Expectations from the course

Prerequisites 🎓

You should have experience with…

  • … the Linux operating system 🐧
  • … the Bash shell (interactive, scripting) 🐚
  • … using the Secure Shell (SSH) 🛡️
  • … one programming language (ideally: Python 🐍)
  • … (some exposure to) scientific programming / machine learning

Also: you need an account on the BIH HPC 🔑

What is High-Performance Computing?

  • Attempt at a definition
  • Role of HPC in biomedical and medical research
  • Trade-Offs

Attempt at a Definition: HPC …

  • refers to advanced computing techniques & technologies to solve complex computational problems efficiently
  • involves leveraging parallel processing, large-scale data analysis, and specialized hardware
    • to achieve high computational performance
  • systems consist of multiple computing nodes connected through a high-speed network, working together
  • enables researchers to tackle computationally intensive tasks that would be infeasible or too time-consuming otherwise
  • finds applications in a wide range of fields, including scientific research, engineering, data analytics, and machine learning

HPC in Biomedical Research

  • … plays a crucial role by enabling tackling of computational challenges
  • … allows for analyzing large-scale genomics, proteomics, …, datasets
    • leading to insights into diseases and potential treatments
  • … facilitates simulations such as protein folding, molecule interactions, etc.
  • … enables the efficient training of large-scale statistical and machine learning models

Trade-Offs of HPC

Advantages

  • fast execution of complex computational tasks
  • process and analyze large data sets
  • fast and large storage systems
  • MORE POWER 🦾

Drawbacks

  • learning curve / entry barrier
  • usually shared with other users
  • expensive to buy/operate
  • high power usage/CO2 footprint (reference)
  • “why is my job killed/crashing/not running?” 😶‍🌫️

There is no free lunch!

What is Your Take? 🤸

“Blitzlicht” 📸

  • answer one of the questions
  • do not repeat a previous answer

So far

  • have you benefited from advantages?
  • have you suffered from drawbacks?

From here on

  • what do you hope to gain from using HPC?
  • what risks do you see?

Programming HPC Systems

  • There is no Magic or Silver Bullet
  • Common Paradigms for Parallel Programing

⚠️ “Warning”: just a quick and superficial overview ;-)

There Is No Magic (1)

Cray-1 1976 @80Mhz Wikipedia

IBM Blue Gene ~2000-2015 Wikipedia

“Just a (large) bunch of computers with fast interconnect.”

There Is No Magic (2)

  • From the perspective of 1976, your smartphone is a supercomputer.
  • iPhone 14
    • ~1,000EUR
    • 6 cores @3.23Ghz; 6GB RAM
    • GPU: 5-core GPU
    • “Neural Engine”: 15.8TOPs
  • 2002 NEC Earth Simulator
    • 412,000,000EUR (2023; inflation corrected)
    • 36TFLOPs
  • 1976 Cray-1
    • 24,623,570EUR (2023; inflation corrected)
    • 160MFLOPS

There Is No Magic (3)

  • Is it easy to program an iPhone?
    • “yes and no”
  • Is it easy to program for an HPC?
    • “yes and no”

“Es gibt keinen Königsweg” (1)

aka. “There is no silver bullet”.

Steps to mastery:

  • Master using Linux.
  • Master programming for
    • a single core
    • multiple cores on a single machine
    • multiple cores on multiple machines
    • programming GPUs
  • Realize that most problems are

“Es gibt keinen Königsweg” (2)

aka. “There is no silver bullet”.

Steps to mastery:

  • Master using Linux.
  • Master programming for
    • a single core
    • multiple cores on a single machine
    • multiple cores on multiple machines
    • programming GPUs
  • Realize that most problems
    • are embarassingly parallel

“Es gibt keinen Königsweg” (3)

aka. “There is no silver bullet”.

Steps to mastery:

  • Master using Linux.
  • Master programming for
    • a single core
    • multiple cores on a single machine
    • multiple cores on multiple machines
    • programming GPUs
  • Realize that most problems
    • are embarassingly parallel
    • have well-solved “cores”

Types of Parallelism

Single-core level (no focus here)

  • Bit-level parallelism
    • aka bit-wise AND, OR, etc.
  • Instruction-level parallelism
    • aka instruction pipelining

Programming level

  • Pipeline parallelism
  • Task parallelism
  • Data parallelism

Pipeline Parallelism

Source: Stanford

  • Consider an imaginary laundry
    • Laundry steps: wash, dry, ironing
    • One machine/operatore per step
  • Naive / sequential execution is slow
  • We can speedup the process with pipeline parallism
    • The bottleneck / dominating step is drying
    • Also: pipeline startup/shutdown

Task Parallism (1)

  • Often, work can be split into different tasks
    • e.g., one task per input file
  • These tasks can be processed independently
  • Tasks may have different “sizes” (time for processing, required RAM, …)
  • Tasks may have dependencies (output of one task is input for another)
  • If we can easily split (part of) the work into independent tasks:
    • embarassingly parallel

Task Parallism (2)

  • How to process these tasks?
  • Commonly, the manager-worker pattern is applied.
  • The manager
    • has a (potentially dynamic) list of tasks
    • distributes the tasks to the workers
    • the workers do the actual work

Data Parallism

  • specialized hardware examples: GPUs & TPUs
  • Regularly structured data (e.g., vectors, matrices, tensors) …
    • … have natural decompositions …
    • … and operations can be easily parallelized.
  • Common applications:
    • Linear algebra / graphics processing
    • “Deep” learning etc.
    • Finite element methods for differential equations
      • (but with a twist!)

HPC Systems and Architecture

  • Compute nodes
  • Shared memory vs. distributed memory systems
  • Cluster architecture
  • Distributed file systems
  • Job schedulers and resource management

⚠️ “Warning”: just a quick and superficial overview ;-)

Compute Nodes (1)

“Same-same (as your laptop), but different.”

  • 2+ sockets with
    • many-cores CPUs
    • e.g., 2 x 24 x 2 = 96 threads
  • high memory (e.g., 200+ GB)
  • fast network interface card
    • “legacy”: 10GbE (x2)
    • modern: 25GbE (x2)
  • local disks
    • HDD or solid state SSD/NVME

Compute Nodes (2)

More differences from “consumer-grade” hardware:

  • error correcting memory (bit flips are real)
    • Google in 2009: 8% of DIMMs have 1+ 1bit errors/year, 0.2% of DIMMs have 1+ 2bit errors/year
  • stronger fans
  • redundant power control
  • redundant disks

You are not the admin

no root/admin access, no sudo

Shared vs. Distributed Memory

Shared Memory

  • in-core/multi-threading
graph BT
    sq1[thread 1] --> ci(memory address)
    sq2[thread 2] --> ci(memory address)
  • ➕ low overhead
  • ➕ easy to get started
  • ➖ implicit communication, easy to make errors
  • ➖ do you really understand your memory model?

Distributed Memory

  • out-of-core/message-passing
graph LR
    sq1[thread 1] -->|How are you?| sq2[thread 2]
    sq2[thread 2] -->|- fine, thank!| sq1[thread 1]
  • ➕ explicit communication, fewer wrong assumptions(?)
  • ➕ model scales up better for larger systems
  • ➖ harder to get started
  • ➖ more complex primitives

Your Experience? 🤸

“Blitzlicht” 📸

  • answer one of the questions
  • do not repeat a previous answer
  • have you used shared/distributed
    memory parallelism before?
  • what is your experience/hope?

Cluster Architecture

  • head nodes (login/transfer)
  • compute nodes
    • generic: cpu
    • specialized: high-mem/gpus
  • storage cluster with parallel file system
  • scheduler to orchestrate jobs
  • Network/Interconnect

(Distributed) File Systems

  • directory/files correspond to inode
    • inode: fixed-size data store
  • special inodes such as root directory
    • “it’s all inodes from there”
  • large directories:
    • indirection
    • OS must scan all indirect blocks for an ls
  • More? Ext4 Disk Layout

Source: Wikipedia

Distributed File Systems (2)

“Same-same (as your laptop), but different.”

  • POSIX file system
    • laptop: ext4/XFS/btrfs/ZFS/…
    • distributed: CephFS, GPFS/SpectrumScale, BeeGFS, …
  • POSIX guarantees
    • sync() → visible everywhere
    • mkdir/open() → visible everywhere
    • files can be opened by multiple processes
  • … harder to enforce in a distributed (multi-node) setting

Distributed File Systems (3)

Best Practices / Do’s & Don’ts

  • use modest sized directories (<10k entries)
    • don’t create one file per gene (or similar)
    • create sub directories, e.g., abcdef → ab/cdef
  • don’t splurge in file count
    • don’t create one file per NGS read (or similar)
  • avoid recursive traversal of large structures
    • ls -lR will be slow!

Distributed File Systems (4)

Best Practices / Do’s & Don’ts

  • avoid small reads/writes and random access
    • each I/O operation (IOP) needs to go through the network
    • I/O systems are better at handling larger/sequential reads/writes
  • DO stream through your files
    • for each line/record in file: # do work
  • DO use Unix sort
  • DO use Unix pipelines rather than temporary files
    • e.g., seqtk mergepe R1.fq R2.fq | bwa mem ref.fa | samtools sort | samtools view -O out.bam

Distributed File Systems (6)

Best Practices / Do’s & Don’ts

  • DON’T be afraid of large files
    • 1GB does not qualify as large
    • 1TB files … do

Job Scheduler and Resource Management

sequenceDiagram
    autonumber
    User-)+Scheduler: sbatch $resource_args jobscript.sh
    Scheduler->>+Scheduler: add job to queue
    User-)+Scheduler: squeue / scontrol show job
    Scheduler-->>+User: job status
    Note right of Scheduler: scheduler loop
    Scheduler-)Compute Node: start job
    Compute Node->>Compute Node: execute job
    Compute Node-)+Scheduler: job complete

Last but not least…

We should make sure that you have access to the HPC system…

Connecting to the HPC

  • Windows Users
  • Linux / Mac Users
    • Should be able to do `ssh -l $USER_{c,m} hpc-login-1.cubi.bihealth.org

Further Sessions

  1. Overview (this)
  2. Slurm Job Scheduler and Resource Manager
  3. Scientific Programming with Python
  4. Reproducible Workflows with Snakemake

This is not the end…

… but all for the first session

Recap

  • preqrequites
  • HPC in general and biomedical sciences
  • HPC hardware and cluster architecture
  • distributed file systems
  • a first peek at job schedulers
  • further sessions

What’s next?

Course description: “where participants can apply their learnings in the context of their PhD projects”

How this applies to this session:

  • We ensure that your HPC access works
  • We will review your current settings (one on one)
  • Let’s discuss productivity:
    • places (online) to find documentation / get answers
    • “sleight of hand” ♥️♠️♣️♦️
    • text editors
    • Bash features: shortcuts, history, …

Lab Tasks

Please spend the rest of the day setting up or learning about …